Multi-tenancy promises high utilization of available system resources and helps maintain cost-effective operations for service providers. However, multi-tenant high-performance computing (HPC) infrastructures bring unique challenges, both associated with providing performance isolation to the tenants, and achieving efficient load-balancing across the network fabric.